Skip to content

PERF: to_datetime with uint #43268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 30, 2021
Merged

PERF: to_datetime with uint #43268

merged 2 commits into from
Aug 30, 2021

Conversation

debnathshoham
Copy link
Member

@debnathshoham debnathshoham commented Aug 28, 2021 •
edited
Loading 

#PR
In [3]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint64"), unit="s", utc=True)
CPU times: user 20.6 ms, sys: 19.9 ms, total: 40.5 ms
Wall time: 39.2 ms

In [4]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint32"), unit="s", utc=True)
CPU times: user 16.3 ms, sys: 6.58 ms, total: 22.8 ms
Wall time: 21.2 ms

#master
In [3]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint64"), unit="s", utc=True)
CPU times: user 12.6 s, sys: 44.3 ms, total: 12.7 s
Wall time: 12.7 s

In [4]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint32"), unit="s", utc=True)
CPU times: user 5.88 s, sys: 21.2 ms, total: 5.9 s
Wall time: 5.92 s

@simonjayhawkins simonjayhawkins added Performance Memory or execution speed performance Datetime Datetime data dtype labels Aug 29, 2021
@jreback jreback added this to the 1.4 milestone Aug 30, 2021
@jreback jreback merged commit 303fc9a into pandas-dev:master Aug 30, 2021
@jreback
Copy link
Contributor

jreback commented Aug 30, 2021

thanks @debnathshoham very nice

@debnathshoham debnathshoham deleted the gh42606 branch August 30, 2021 13:48
feefladder pushed a commit to feefladder/pandas that referenced this pull request Sep 7, 2021
@debnathshoham @feefladder
PERF: to_datetime with uint (pandas-dev#43268)
9a1c6c1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Datetime Datetime data dtype Performance Memory or execution speed performance
Projects
None yet
Milestone
1.4
Development

Successfully merging this pull request may close these issues.

BUG: to_datetime very slow with unsigned ints for unix seconds
3 participants
@debnathshoham @jreback @simonjayhawkins